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1 . INTRODUCTION 


Label Identification from Statistical Tabulation (LIST) Is an analyst- 
interpreter (AI) picture-element (pixel) labeling procedure for making at- 
harvest percentage small -grain estimates In the Large Area Crop Inventory 
Experiment (LACIE)J In this labeling procedure, the AI is required to 
answer questions about the segment and pixels which relate to simple 
properties that discriminate small grains from nonsmall grains. The responses, 
along with pertinent agricultural and meteorological variables, are statis- 
tically weighted to develop a discriminant function which is trained on 
blind-site ground-truth labels. 

Results from an ear'ier development of LIST were analyzed and reported by 
Pore in November 1977 (ref. 1). Those results were used to develop a semi- 
automated, operational LIST reported by Abotteen and Pore in February 1978 
(ref. 2). This newly developed operational LIST was tested on both Kansas 
and North Dakota blind sites, and the results of those tests are reported here. 

Section 2 describes the analyses performed, and section 3 gives the results 
of those analyses. An evaluation and recommendations follow in section 4. 


The LACIE is a joint undertaking of the U.S. Department of Agriculture (USDA), 
the National Oceanic and Atmospheric Administration (NOAA) of the U.S. Depart- 
ment of Commerce, and the National Aeronautics and Space Administration (NASA). 
The procedures which are the subject of this paper were developed at the NASA 
Lyndon B. Johnson Space Center (JSC) for the Earth Observations Division (EOD), 
Space and Life Sciences Directorate. 
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2. ANALYSES 


Four AI's were used to test the quality of the questions for discriminating 
small grains (agricultural crops) from nonsmall grains. Each AI analyzed 
16 segments, taking approximately 2-1/2 hours per segment. Each set of 
22 932 pixels in a given area is referred to as a segment and covers a 
rectangular area of approximately 9 by 11 kilometers (5 by 6 nautical miles). 

Every pixel at the intersection of a 10-by-10 grid is a grid pixel (or grid 
dot). Two hundred nine grid dots are in each segment, and all (and only grid 
dots) were used in this study. An earlier investigation by Register and 
Hocutt (ref. 3) has indicated that interpixel correlations decrease with 
distance and that a distance of 10 pixel widths corresponds to negligible 
correlation. Hence, dot grids are assumed to be independent samples with 
respect to crop types. 

Separate analyses were performed for the 1976 winter and spring wheat sites, 
there being eight of each. All Kansas blind sites with available ground 
truth in stratum 11 of the New LACIE Strata were chosen as the winter wheat 
test sites. The eight spring wheat sites were chosen from the blind sites 
in stratum 21 (figure 1 shows locations of New LACIE Strata). Since ground 
truth was required in stratum 21, segments were chosen to be representative 
of the three-state coverage of the stratum. The data within each stratum 
were further partitioned into four training and four test segments (table 1). 

For each segment, four acquisition dates were chosen arbitrarily without 
respect to special areal agricultural -meteorological conditions such as 
cloud cover, etc.; these were chosen to cover generally the 1975-76 growing 
season for wheat. Table 2 gives these dates and the respective Robertson 
biostages for winter wheat and spring wheat. Three types of production film 
converter (PFC) products were generated: type 1, type 2, and the Kraus 

product (see reference 4, Austin, for a description of these films). The 
films were made into Research, Test, and Evaluation (RT&E) packets and kept 
separate from LACIE operational packets. This was done to maintain a 
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TABLE 1 LIST DATA SET 


Stratum 

LAC IE 
segment 

Type 

kL 

Purpose 

County 

State 

11 

1019 

ww 

Training 

Norton 

Kans. 

11 

1035 

ww 

Training 

Ford 

Kans. 

11 

1855 

ww 

Training 

Trego 

Kans. 

11 

1865 

ww 

Training 

Stevens 

Kans. 

11 

1020 

ww 

Test 

Rawlins 

Kans. 

11 

1852 

ww 

Test 

Lant 

Kans. 

11 

1860 

ww 

Test 

Hodgeman 

Kans. 

11 

1880 

ww 

Test 

Ellis 

Kans. 

21 

1542 

SW 

Training 

Roosevel t 

Mont. 

21 

1650 

SW 

Training 

Hettinger 

N. Dak. 

21 

1651 

SW 

Training 

Bowman 

N. Dak. 

21 

1667 

SW 

Training 

Harding 

S. Dak. 

21 

1530 

SW 

Test 

Phillips 

Mont. 

21 

1656 

SW 

Test 

Morton 

N. Dak. 

21 

1660 

SW 

Test 

Logan 

N. Dak. 

21 

1668 

SW 

Test 

Perkins 

S. Dak. 


a WW = winter wheat; SW * spring wheat. 
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TABLE 2.- LIST DATA ACQUISITIONS (1976) 


1 

1 mm 

County 

Date 

Biostage 

WW 

SU 

1019 

Norton 

Jan. 19 

2.4 




Feb. 6 

2.5 




June 12 

4.6 




June 30 

5.4 


1020 

Rawl ins 

Feb. 25 

2.5 




Apr. 10 

2.7 




June 3 

3.7 




July 18 

6.0 


1035 

Ford 

Mar. 13 

2.6 




May 6 

3.4 




June 1 

4.1 




July 8 

6.0 


1530 

Phillips 

June 1 

3.5 

3.1 



June 18 

4.0 

3.9 



July 7 

5.5 

5.0 



Aug. 12 

7.0 

6.0 

1542 

Roosevelt 

Apr. 25 

2.5 

1.1 



June 18 

4.3 

3.4 



July 6 

5.7 

5.0 



July 24 

6.0 

6.0 

1650 

Hettinger 

May 9 

3.2 

2.0 



May 27 

3.8 

3.0 



Aug. 7 

6.0 

6.0 



Aug. 25 

6.0 

6.0 

1651 

Bowman 

May 10 

3.3 

2.2 



May 29 

4.0 

3.0 



July 21 

6.0 

6.0 



Aug. 8 

6.0 

6.0 


2-3 


4 






TABLE 2.- Continued. 


LAC IE 
segment 

County 

Date 

Biostage 

ww 

SW 

1656 

Morton 


3.0 

2.0 



WBSM 

6.0 

4.4 



m 

7.0 

6.0 



Aug. 7 

7.0 

7.0 

1660 

Logan 

May 7 

3.1 

2.0 



June 12 

4.2 

3.7 



Aug. 6 

6.0 

6.0 



Aug. 23 

6.0 

6.0 

1667 

Harding 

May 10 

3.4 

2.3 



May 29 

4.3 

3.2 



July 21 

6.0 

5.9 



Aug. 8 

6.0 

6.0 

1668 

Perkins 

Apr. 22 

2.6 

1.7 



May 9 

3.3 

2.3 



May 28 

4.0 

3.1 



Aug. 7 

6.0 

6.0 

1852 

Lane 

Mar. 31 

2.6 




May 7 

3.2 




June 20 

5.8 


. 


July 17 

6.0 


1855 

Trego 

Mar. 13 

2.6 




Apr. 18 

3.0 




June 20 

5.7 




July 17 

6.0 
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TABLE 2.- Concluded. 



County 

Date 

Hodgeman 

Mar. 13 
May 6 
June 2 
July 8 

Stevens 

Feb. 7 
May 15 
June 20 
July 8 

Ellis 

Mar. 13 
May 6 
June 10 
July 16 


Biostage 
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Figure 1.- Map of U.S. Great Plains showing New LACIE Strata 



restricted experimental environment of labeling without a 9- by 9-1nch film 
Image (covering a 185- by 185-kilometer track of land) of the broad area of 
Interest and without ancillary agricultural-meteorological Information. 

Hence, accuracies should be below those experienced In an operational labeling 
system. 

The LIST procedure consists of obtaining AI responses to a set of questions 
directed at describing simple properties of the grid dots. The format used 
is presented in appendix A. These responses directly yield three categories 
or labels for the pixels. 

a. Column 2 determines a designated other (DO) category. 

b. Columns 3, 4, and 5 determine a nonclassiflable category. 

c. The balance, those for which columns 6 through 9 are answered, constitutes 
a category of "pure" or labelable pixels. 

The border pixels were omitted from the study, and their disposition will 
be discussed in section 4. The DO pixels were not part of the analysis or 
discriminating process but are reported as LIST results. This is because 
the LIST procedure (as presently defined) accepts the AI designation of DO 
as a LIST label. Only the pixels which could be labeled were admitted 
into analysis. This minimizes the effect of outliers and unlabelable pixels, 
thus producing more precise labeling functions. 

The first analysis consisted of a stepwise linear discriminant analysis 
using the Statistical Package for the Social Sciences (SPSS, ref. 5). The 
major options were to base prior probabilities of category membership on 
training sample sizes and to use the minimum residuals method of stepping 
variables in and out of discrimination. Other analyses were a direct dis- 
criminant analysis that automatically uses every variable under consideration 
and a quadratic discriminant procedure that includes a*l linear terms and 
all two-way products (including squared terms). This latter procedure 
utilized the Patterson-Pltt algorithm as implemented by Thadani (ref. 6) 
and Ahlers (ref. 7). The discriminants were determined using ground truth 
on the four training segments, and accuracy was determined using the 
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discriminant function to classify the four test segments. Percentages of 
pixels correctly labeled were calculated from contingency tables of ground 
truth by LIST. 

Ar. SPSS program listing for the spring wheat site LIST Is given In appendix B 
as representative documentation of the automation process. 
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3. RESULTS 


The particular variables admitted by a stepwise discriminant procedure are 
the number of training samples, the variability of the particular area 
sampled, acquisition dates, etc. Certainly, It Is not recommended that a 
training sample of the size used here be Implemented In LAC IE; hence, 
discriminant vectors and tests for category mean differences will not be 
presented here. Instead, tables for test accuracy (on segments not used In 
training) are presented. Table 3 Is a key to these contingency tables. 

Four analyses were performed on the winter wheat segments: two using the 
quadratic discriminator (Q), one using the stepwise discriminant, and one 
using the AI labels. Table 4 gives these results for all four AI‘s, each 
responding to the four winter wheat test segments. Table 5 lists the 
variables used In the respective parts of table 4. Appendix C gives 
variable definitions for all analyse. As presently programmed, the 
quadratic discriminator was determ ’sr! to accrue numerical analysis errors 
of computation at an unacceptable rate and was not used in the spring wheat 
site analyses. 

All spring wheat sites were treated as mixed-wheat sites, even where winter 
wheat analysis was patently unnecessary. The mlxed-whiat philosophy was to 
give positive responses automatically where Indicated for either spring or 
winter wheat. For example, if the canopy trajectory for a pixel is similar 
to a winter wheat trajectory (SUM is high for winter wheat biostage numbers) 
while It is dissimilar for spring wheat (SUM is low for spring wheat biostage 
numbers), then KEYS and SUM are based on winter wheat biostages for that 
pixel. Tables 6 and 7 give the results for the spring wheat sites. 

The AI percentage of small grains and the LIST percentage of small grains 
were consistently below the ground-truth percentage of small grains (m < l 
in table 3), regardless of the type of discriminant used. Inis is partially 
attributed to the fact that (1) omission rates apparently are always less 


TABLE 3.- CONTINGENCY TABLE KEY 


Type of labeler 



SG 

Non 

SL 

SG 

a 

b + e 

g 

Non 

c 

d + f 

h 

m 

i 

j 

k 


PCL 


Variable 


Definition 


a, b, c, d 
e, f 

g, h, i, j 

9 

GT 

h 

i 

j 

k 

£ 

m 

Non 

PCL 

SG 


Raw pixel counts for the four test segments 
Raw pixel counts for the DO pixels 
Marginal probabilities (expressed as percentages) of 
correct labeling (PCL's): 

a + b ~~ + e * 100 = ^ " Momission)] x 100 


Ground truth 

c +~ d ~ T~T x ^0 = [1 - Pr(commission)] * 100 


a + c 


x IUU 


d + f 


x 100 


b + d + e + f 
a+b+c+d+e+f 

a — x loo = ground- truth percentage of small grains 


-~ Y c x 100 = LIST labeled percentage of small grains 
Nonsmall grains 

9 + k * ~ x ^ = P ro ^ability (expressed as a percentage) 
of correct labeling 

Small grains 
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TABLE 5 - LIST TEST VARIABLES FOR WINTER WHEAT SITES 


Title 
AI labels 

Linear discriminant 


Q with B&G only 


Q17 


Variables 

Analyst label. 

Gl, canopy trajectory, B4, GREEN3, B2, G4, KEY4, B1 , 
G2, PCGW, G3, KEY3, GREEN2, KEY 2, BI04, GREEN4, 

BI02. 

Bl, B2, B3, B4, Gl , G2, G3, G4, and all possible 
interactions. 

BI02, BI04, Bl, B2, B4, Gl , G2, G3, G4, GREEN2, 
GREEN3, GREEN4, PCGW, KEY2, KEY 3, KEY4, canopy 
trajectory, and all possible interactions. 


TABLE 6.- LIST TEST ACCURACY ON SPRING WHEAT SITES 


AI label 


Linear with B-G-BIO step 



SG 

Non 

8.8% 

SG 

113 

32 + 79 

50% 

Non 

47 

166 + 2106 

98% 

6.3% 

71% 

95% 

2543 



SG 

Non 

8.8% 

SG 

105 

40 + 79 

47% 

Non 

47 

166 + 2106 

98% 

6.0% 

69% 

95% 

2543 


Linear discriminant 


Linear with B-G-BIO direct 



SG 

Non 

8.8% 

SG 

105 

40 + 79 

47% 

Non 

67 

146 + 2106 

97% 

6.8% 

61% 

95% 

2543 



SG 

Non 

8.3% 

SG 

100 

45 + 79 

45% 

Non 

41 

172 + 2106 

98% 

5.5% 

71% 

95% 

2543 



























TABLE 7.- LIST TEST VARIABLES FOR SPRING WHEAT SITES 


Title 

AI labels 

Linear discriminant 
Linear with B-G-BIO step 

Linear with B-G-BIO direct 


Variables (in order of inclusion) 

AI label. 

Canopy trajectory, G1 , G3, B4, B1 , GREEN1 , 
G2, G4, GREEN4, PCGW, B3, KEY3, B2. 

Canopy trajectory, GS1 , GW3, BS4, BW1 , 

GREEN! , GW1 , GS4, GS2, GREEN4, BS3, KEY3, 
PCGW, BW3, BS2. 

GW1, GW2, GW3, GW4, GS1 , GS2, GS3, GS4, BW1 , 
BW2, BW3, BW4, BS1 , BS2, BS3, BS4, PCGW, 
PCGS, canopy trajectory, KEY! , KEY2, KEY 3, 
KEY4, GREEN 1 , GREEN2, GREEN3, GREEN4. 



than commission rates (b < c in table 3) and (2) a fairly consistent 
tendency exists for nearly 4 percent of the DO pixels to be small grains 

(eT7 * °’ 038 )- 

Mid-season estimation cannot be analyzed effectively because (1) acquisition 
date selection for end-of-season estimation is usually inappropriate for 
mid-season estimation and (2) specialized mid-season questions (e.g., automated 
prototype green number trajectories) have not been developed. Nevertheless, 
such an analysis is presented here, recognizing that lower than realistic 
accuracy is expected. Such an analysis indicates the efficacy of present 
keys and may be of heuristic value in pointing to new developments. A 
rather high accuracy (PCL in the terminology of table 3) and a moderate 
decrease in the percentage of small grains reported (m < i in the terminology 
of table 3) are demonstrated in tables 8 and 9. 
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TABLE 8 - MID-SEASON TEST ACCURACY 


#> 


GT 


Winter sites 



SG 

Non 

21.1% 

SG 

409 

117 + 65 

69% 

Non 

113 

546 + 1553 

95% 

18.6% 

78% 

92% 

2803 


PCL = 89.5% 


Spring sites 



SG 

Non 

8.8% 

SG 

85 

60 + 79 

38% 

Non 

38 

175 + 2106 

98% 

4.8% 

69% 

94% 

2543 


PCL = 93.0% 


TABLE 9.- MID-SEASON TEST VARIABLES 


— 

Title 

Variables 

WW sites 

BI01 , BI02, Gl, Bl, G2, B2, KEY2, GREEN!, GREEN2, GUI , GW2, 
BW2. 

SW sites 

SB 1 01 , SBI02, WBI01 , WBI02, GWl , GW2, BW1 , BW2 , GS1 , GS2 , 
BS1 , KEY1 , KEY2, GREEN!, GREEN2. 
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4. EVALUATION AND RECOMMENDATIONS 


The inclusion of all possible interactions, as is accomplished routinely 
using the Patterson-Pitt quadratic discriminator, does not appear to increase 
classification accuracy because of the inclusion of too many spurious 
variables. However, the selective construction of greenness/biostage and 
brightness/biostage interactions does appear to raise the PCL and could be 
Incorporated beneficially in succeeding LIST developments. 

The phenomenon of nearly 4 percent DO being small grains constitutes a 
source of bias that is apparently consistent over diverse geographic regions 
and is readily measurable. A study to measure this bias and develop a bias- 
correction procedure would be beneficial in the development of an operational 
LIST system. 

The unexpectedly high PCL (high means close to AI labeling accuracy) in the 
"undeveloped discriminator" for mid-season labeling analyses suggests that 
directed development of a mid-season LIST labeler (as opposed to a causal 
byproduct of an end-of-season LIST labeler) would yield a highly accurate 
operational labeling system. 

The present Classification and Mensuration System (CAMS) procedural philosophy 
is for the AI to select imagery for a "reference" acquisition date and 
mentally adjust the registration discrepancies of other acquisitions to 
give accurate labels to the "real estate" represented in the reference film. 

It is becoming increasingly evident that LIST, and in fact any labeling 
procedure that relies on spectral aids (e.g., trajectories), is inherently 
based on a different philosophy. Since acquisitions are usually not registered 
identically, spectral values for a pixel across several acquisitions there- 
fore represent the area about the "real estate" and not a precise pixel of 
one date. Boundary pixels and mixed pixels (across a boundary) have spurious 
spectral trajectories; i.e., the trajectory is not sampled from any category 
of interest but switches from one category to another. Such trajectories 
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tend to confuse the labeling process and reflect a basic modeling error In 
Image Interpretation. LIST, on the other hand, labels what Is represented 
by the trajectory, which, in this case, is the grid dot intersection on the 
PFC product. To make this more meaningful, LIST first filters out the 
boundary (and mixed) pixels and then treats these pixels as a nonlabelable 
class to be proportioned. In summary, LIST does not label real estate but 
does label film grid intersection pixels. This philosophical change is 
implied by the increased reliance on spectral trajectories. 

The high accuracies in tables 4 and 6 demonstrate that the concept of a 
statistical discrimination approach to pixel labeling is a valid concept 
and, in particular, that the LIST procedure (appendix A) performed comparably 
with AI methods. In the restrictive environment of these test conditions, 
this is a highly successful result that confirms the efficacy of this LIST 
questionnaire. However, it can be easily and obviously improved through 
the development and training of the automated keys, and particularly green 
number ranges and trajectories. 

The recommendations made by Abotteen and Pore in February 1978 (ref. 2) are 
still applicable and can be expanded as follows: 

a. A stratified estimation procedure for using LIST in LAC IE Procedure 1 
area estimation, where permissible labels are small-grains, other, and 
boundary pixels, should be developed. 

b. A set of suitable questions for discrimination of wheat from other 
small grains could be profitably developed. 

c. An early-season technology of LIST labels could be developed easily from 
the LIST developments represented here. 

d. A multicrop (corn/soybean) LIST technology is certainly indicated from 
the small -grain/other successes reported here. 

e. Adaptation of this LIST to an interactive color console computer system 
would advance pixel labeling technology to a cybernetic (feedback) 
process that could increase accuracy and possibly decrease operational 
processing time. 
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APPENDIX A 


LIST QUESTIONNAIRE 


Line 1 of Keypunch Transmittal Form 
Column 


Entry 


1-5 

6-30 

31-33 

34 


36-40 



Segment number. 

County, state, or country If not United States. 

Universal strata number. 

Segment type: 

1 - Winter wheat. 

2 - Mixed wheat. 

3 - Spring wheat. 

Acquisition date chosen by analyst as registration date 
(YDDD). (This is not necessarily the Goddard Space Flight 
Center reference segment.) 


Interpretable acquisition dates (YDDD). 


Line 2 of Keypunch Transmittal Form 


Column 

1-5 


7-8 

10-11 

13-14 

16-17 

19-21 


Entry 

Segment number. 

Sun angles for the respective acquisitions. 


Robertson winter wheat biostages. 


A-l 




Column 


Entry 


23-25 \ 

27-29 > Robertson winter wheat biostages (continued). 

31-33 ) 

35-37 \ 

39-41 ( 

> Robertson spring wheat biostages. 

43-45 I 

47-49 ) 

Succeeding lines of Keypunch Transmittal Form 
Column Entry 

1 Leave the first column blank. 

2 1 - Pixel is In nonagrl cultural area. STOP; pixel Is DO. 

Go to 9. 

0 or blank - Agricultural area or Indeterminate. 

3 Is pixel registered with regard to analyst chosen registration 
date (l.e.. In the same category)? 

1 - No. STOP; pixel Is not classifiable. Go to 9. 

0 or blank - Yes or Indeterminate. 

4 Is pixel a mixed pixel (part of more than one field or 
boundary)? 

1 - Yes. STOP; pixel Is not classifiable. Go to 9. 

0 or blank - No or Indeterminate. 

5 Is this an anomalous pixel (not representative of most of 
the other pixels within the field)? 

1 - Yes. STOP; pixel Is not classifiable. Go to 9. 

2 - No. 


A-2 


P 


Column Entry 

6-9 PFC vegetation canopy Indication Is 

(Use all available imagery film typesTJ 

0 - No vegetation canopy. 

1 - Low-density green vegetation canopy. 

2 - Medium-density green vegetation canopy. 

3 - High-density vegetation canopy. 

4 - Senesclng (turning) vegetation canopy. 

5 - Harvested canopy (stubble). 

11-14 Is the vegetation indication of the pixel on PFC Imagery 
valid for the Robertson biostage of wheat for the 
acquisition? (Check keys for partition.) 

1 - No. 

2 - Yes. 

15 Pixel is: 

1 - Small grains. 

2 - Other. 

16-18 Line (or row) number of pixel. 

20-22 Column number of pixel. 
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AUTOMATED LIST QUESTIONS FOR SMALL-GRAINS CLASSIFICATION 


Green number of pixel is . (Corrected to 

60° latitude.) 

Is the green number of the pixel within the range for small grains? 

Yes 

No 

Brightness number of pixel is . 

The winter principal component greenness (PCG) statistic is 

The spring PCG statistic is . 

Is the vegetation indication of the pixel valid for the Robertson 
biostage of wheat for the acquisition? 

Yes 

No 

Does the pixel follow a small-grains spectral development pattern? 

Yes 

No 
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VARIABLE DEFINITIONS FOR ANALYSES 3 


Variable 

B-G-BIO 

BIOT, BI02, BI03, BI04 
or 

WBIOl through WBI04 
SB 101 through SBI04 
Gl, G2, G3, G4 
Bl, B2, B3, B4 
GREEN1 through GREEN4 

KEY1 through KEY4 
Canopy trajectory 

PCGW, PCGS 

GW1 through GW4 
GS1 through GS4 
BW1 through BW4 
BS1 through BS4 


Definition 

Brightness, greenness, and biostage interaction 

Winter wheat Robertson biostages for the respective 
acquisitions. 

Spring wheat biostages. 

Green numbers. 

Brightness numbers. 

Yes/No answer: Is green number in the small -grain 

range? 

Yes/No answer: Is canopy in the small-grain range? 

Yes/No answer: Is canopy trajectory acceptable for 

small grains? 

Principal component greenness statistic for winter 
and spring wheat, respectively. 

Products of G i * WBI0 i for i = 1,2, 3, 4. 

Products of G.j * SBICh for i = 1,2, 3, 4. 

Products of B.j * WBIO. for i = 1,2, 3, 4. 

Products of B.j * S B 1 0 ^ for i = 1,2, 3, 4. 


a See Abotteen and Pore (ref. 2) for the numerical derivations. 
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